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Software  Engineering  Institute 

Purpose  of  this  Presentation 

To  offer  a  draft  set  of  TRL  descriptions  for  use  in 
assessing  practice-based  technologies  (PBTs) 

To  outline  the  next  steps  by  which  these  descriptions  will 
be  prototyped,  piloted,  and  tested 
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What  are  PBTs? 


Practices 
Processes 
Methods 
Approaches 
Frameworks  (for 
the  above)  J 


Versus  non-PBTs: 
Hardware 
Software 

Embedded  systems 
Biomedical  devices 
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Product  Line  Practices 
CM  Ml  (framework) 
Acquisition  practices 
Transition  processes 
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DoD  Technology  Readiness  Levels 

A  scale  from  1  to  9  used  to  assess  technology  maturity* 

•  Basic  principles  observed  and  reported. 

•  Technology  concept  and/or  application  formulated. 

•  Analytical  and  experimental  critical  function  and/or 
characteristic  proof  of  concept. 

•  Component  and/or  breadboard  validation  in  laboratory 
environment. 

•  Component  and/or  breadboard  validation  in  relevant 
environment. 

•  System/subsystem  model  or  prototype  demonstration  in  a 
relevant  environment. 

•  System  prototype  demonstration  in  an  operational 
environment. 

•  Actual  system  completed  and  qualified  through  test  and 
demonstration. 

•  Actual  system  proven  through  successful  mission  operations. 


*DoD  Interim  Defense  Acquisition  Guidebook,  October  30,  2002 
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Why  New  TRL  Descriptions  for  PBTs? 

TRL  users  find  current  description  difficult  to  interpret  for 
non-hardware/system  technologies 

e.g.  software,  medical,  practices 

Army  developed  TRL  descriptions  for  software 

Army  Medical  Research  and  Materiel  Command 
developing  TRL  descriptions  for  biomedical  technologies 

AFRL  (Bill  Nolte)  maturing  a  software  tool  for 
implementing  TRLs 

Study  by  SEI  and  Army  CECOM  in  2002  showed  TRLs 
also  not  readily  applied  to  information  assurance  PBTs 


©  2003  by  Carnegie  Mellon  University 


Version  1.0 


page 


Carra^MeUon 

Software  Engineering  Institute 

Why  Should  I  Care? 

Improvement  of  acquisition  practices  will  require  the 
implementation  of  PBTs 

Knowing  the  “readiness”  of  a  PBT  is  important  to 
managing  its  implementation  risks: 

•  “early”  technologies  may  be  suitable  for  some,  but 
require  additional  investment  (to  mature)  for  others 

•  “mature”  technologies  may  be  suitable  for  some,  but 
offer  no  competitive  advantage  to  others  (because 
everyone  has  access  to  it) 
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Implementation  Risk 
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Our  Approach 


Each  TRL  consists  of 
•  a  Definition,  meant  to  be 
technology-independent 


•  a  more  detailed,  technology- 
dependent  Description 


Basic  principles 
observed  and 
reported 


Lowest  level  of  technology  readiness. 
Scientific  research  begins  to  be  translated 
into  applied  research  and  development. 
Examples  might  include  paper  studies  of  a 
technology’s  basic  properties _ 


Our  approach  was  to  modify  the  Description  for 
each  level,  leaving  the  Definition  as  is. 
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Software  Engineering  Institute 

Caveats 

The  Definitions  are  not  really  technology-independent  (e.g., 
the  term  “breadboard”)  but  for  those  who  want  to  use 
TRLs  to  assess  non-hardware/system  technologies,  they’ll 
have  to  live  with  it  if  they  want  to  be  compliant  with  the 
TRL  scale 

TRLs  are  not  the  only  criteria  that  support  technology 
management,  they  are  just  one  of  numerous  criteria 

Users  in  the  SEI/CECOM  study  estimated  the  TRL  scale 
provides  them  up  to  30%  of  their  decision  criteria 
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Checkpoint 

At  this  point,  you  should  understand 

•  the  importance  of  assessing  PBT  readiness  as  a 
matter  of  managing  implementation  risk 

•  that  current  TRL  descriptions  are  difficult  to  apply  to 
the  PBT  context 

In  the  next  few  slides,  we  show 

•  A  mapping  between  the  TRLs  for  hardware/system 
context  and  our  proposed  TRLs  for  PBTs 

•  an  example  using  SW-CMM 
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TRL  Readiness  Fundamentals  in  the 
Hardware/Systems  Context 

For  hardware/systems,  TRLs  1-9  depict  the  following 
general  progression  in  readiness: 

•  The  environment  in  which  the  technology  can  function 
becomes  more  representative  of  the  final  operational 
environment 

-  from  paper  studies  through  laboratory  setup, 
simulated  environments,  to  mission  operations 

•  The  completeness  of  the  technology  increases 

-  from  basic  properties  through  breadboard 
components,  integrated  components,  prototype,  to 
final  form 
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What  Does  this  Mean  for  PBTs? 

The  environment  in  which  the  technology  can  function 
becomes  more  representative  of  the  final  operational 
environment  (a  community  of  users) 

-  for  PBTs  this  means  the  community  of  users 
expands  from  initial  risk  takers  to  more  mainstream 
members  of  the  community 

The  completeness  of  the  technology  increases 

-  For  PBTs  this  means  the  technology  progresses 
from  defined  basic  properties  through  defined  core 
practices,  implementation  mechanisms,  best 
practices,  to  a  body  of  knowledge 
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Key  Differences 

The  operating  environment  for  PBTs 
is  people/organizations/community, 
not  hardware/systems 

PBT  environment  is  more  mutable, 
malleable,  in  flux 
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PBT  Corollaries  -  draft 


TRL 

HW/System 

PBT 

1 

Scientific  research,  paper 
studies 

Scientific,  behavioral,  and  market  research,  paper 

studies 

2 

Practical,  speculative 
applications  invented 

Practical,  speculative  applications  invented,  potential 
user  communities  identified 

3 

Active  R&D  initiated, 
analytical  and  lab  studies  of 
components 

Active  R&D  initiated,  critical  elements  identified  and 
demonstrated  with  innovative  users 

4 

Basic  components 
integrated,  lab  environment 

Basic  elements  integrated  to  form  core  PBT,  visionary 
leaders  used  to  demonstrate  value  and  transitionability 

5 

Integrated  components 
demonstrated  in  simulated 
environment 

Prototypes  of  implementation  mechanisms  established, 
demonstrated  with  core  PBT  for  pragmatic  users  in 
simulated  environments,  such  as  role-based  workshops 

6 

Prototype  tested  in  relevant 
environment 

Implementation  mechanisms  refined  and  integrated 
with  core  PBT,  demonstrated  in  relevant  environments, 
e.g.,  pilot  settings 

7 

Actual  system  prototype  in 
operational  environment 

Implementation  needs  of  mainstream  users  identified 
and  integrated  into  the  prototype,  operational  use  by 
relevant  users  demonstrated  across  the  community 

8 

Final  form  proven  to  work  in 
operational  environment 

Technology  picked-up  for  wide-spread  rollout  across 
the  community 

9 

Actual  application  running 
under  mission  conditions 

PBT  use  is  considered  routine  within  community,  best 
practices  and  body  of  knowledge  in  place 

-1 
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Example:  SW-CMM 


TRL  # 

Key  Characteristics 

SW-CMM  based  Improvement 
Example 

Nominal 

Timeframe 

1 

Scientific,  behavioral,  and 
market  research,  paper  studies 

IBM  software  framework 
research,  Crosby  research, 
Humphrey  proposal  of  5-level 
maturity  framework 

1985-1987 

2 

Practical,  speculative 
applications  invented,  potential 
user  communities  identified 

Initial  questionnaire 
developed/published  (87-TR-13), 
DoD  and  its  sw-intensive  system 
suppliers  identified 

1986-1987 

3 

Active  R&D  initiated,  critical 

SPA,  87-TR-13  used  with  large 

1987-1989 

elements  identified  and 

DoD  organizations  and 

demonstrated  with  innovative 

contractors;  Managing  the  SI/I/ 

users 

Process  book  published 
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Example:  SW-CMM 


4 

Basic  elements  integrated  to 
form  core  PBT,  visionary 
leaders  used  to  demonstrate 
value  and  transitionability 

SW-CMM  initial  design 
prototyped/tested 

1989-1991 

5 

Prototypes  of  implementation 
mechanisms  established, 
demonstrated  with  core  PBT 
for  pragmatic  users  in 
simulated  environments,  such 
as  role-based  workshops 

SW-CMM  vl.O  published; 
piloted  with  wider  user  base; 
SPA  and  SCE  used  to  feed 
back  info  to  CMM  dev  team; 
SEPG  workshop  becomes 
SEPG  conference 

1991-1993 

6 

Implementation  mechanisms 
refined  and  integrated  with 
core  PBT,  demonstrated  in 
relevant  environments,  e.g., 
pilot  settings 

SW-CMM  vl.1  published;  Intro 
training,  CBA-IPI  and  lead 
appraiser  program  developed; 
ROI  case  studies  published 

1993-1995 
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Example:  SW-CMM 


8 


9 


Implementation  needs  of 
mainstream  users  identified 
and  integrated  into  the 
prototype,  operational  use  by 
relevant  users  demonstrated 
across  the  community 


Technology  picked-up  for 
wide-spread  rollout  across  the 
community 


Transition  Partner,  CBA-IPI,  1993-1997 
SCE  3.0,  Intro  TTT  established; 

SW  measurement  books 
published;  process  support 
(proc  defn,  MPI)  courses 
developed;  SW-CMM  v2.0 
drafted 

“YAMMs”  phenomenon;  high  1995-1997 
maturity  workshops  established; 
principles  for  CMM  established; 

SW-CMM  v2.0  chosen  as  basis 
for  CMMI  framework 


PBT  use  is  considered  routine 
within  community,  best 
practices  and  body  of 
knowledge  are  in  place,  may 
involve  incorporation  of  the 
technology  into  community 
guidance  and  policy 


Incorporation  of  CMM  concepts  1997-2001 
into  ISO  15504;  over  60  orgns 
invited  to  2001  high  maturity 
workshop;  noticeable 
improvement  in  maturity  profile 
for  intended  community;  SW- 
CMM  subsumed  into  CMMI 
(broadening  overall  community) _ 
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Summary  and  Next  Steps 

Initial  draft  of  TRL  Descriptions  for  PBTs  provided 

Community  feedback  and  participation  welcome 

Next  steps  -  pilot  and  test  these  descriptions  with  SEI’s 
and  other’s  PBTs 
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